2025-06-05 19:49:34,438 [ 277109 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2025-06-05 19:49:34,439 [ 277109 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2025-06-05 19:49:34,439 [ 277109 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2025-06-05 19:49:34,439 [ 277109 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_rc7sjw --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_drop_if_empty/test.py::test_drop_if_empty test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper test_failed_async_inserts/test.py::test_failed_async_inserts test_fetch_memory_usage/test.py::test_huge_column test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts test_fix_metadata_version/test.py::test_fix_metadata_version test_freeze_table/test.py::test_freeze_table test_grant_and_revoke/test.py::test_allowed_grantees test_grant_and_revoke/test.py::test_current_database test_grant_and_revoke/test.py::test_current_grants_override test_grant_and_revoke/test.py::test_grant_all_on_table test_grant_and_revoke/test.py::test_grant_current_grants test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke test_grant_and_revoke/test.py::test_grant_option test_grant_and_revoke/test.py::test_grant_with_replace_option test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant test_grant_and_revoke/test.py::test_implicit_create_view_grant test_grant_and_revoke/test.py::test_implicit_show_grants test_grant_and_revoke/test.py::test_introspection test_grant_and_revoke/test.py::test_revoke_requires_grant_option test_grant_and_revoke/test.py::test_smoke test_group_array_element_size/test.py::test_max_exement_size test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 test_http_native/test.py::test_http_native_returns_timezone test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local 'test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0]' 'test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1]' test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement test_insert_over_http_query_log/test.py::test_insert_over_http_ok test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_jbod_ha/test.py::test_jbod_ha test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_availability_zone/test.py::test_get_availability_zone test_keeper_force_recovery/test.py::test_cluster_recovery test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_internal_secure/test.py::test_secure_raft_works test_keeper_internal_secure/test.py::test_secure_raft_works_with_password test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_reconfig_add/test.py::test_reconfig_add test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_s3_snapshot/test.py::test_s3_upload test_keeper_secure_client/test.py::test_connection test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_library_bridge/test.py::test_load_all test_library_bridge/test.py::test_load_all_many_rows test_library_bridge/test.py::test_load_ids test_library_bridge/test.py::test_load_keys test_library_bridge/test.py::test_null_values test_library_bridge/test.py::test_path_validation test_library_bridge/test.py::test_recover_after_bridge_crash test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive test_library_bridge/test.py::test_ssrf test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_lost_part/test.py::test_lost_last_part test_lost_part/test.py::test_lost_part_mutation test_lost_part/test.py::test_lost_part_other_replica test_lost_part/test.py::test_lost_part_same_replica test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: order-1.0.1, random-0.2, timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] test_keeper_four_word_command/test.py::test_cmd_clrs test_lost_part/test.py::test_lost_last_part test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down test_library_bridge/test.py::test_load_all test_log_family_s3/test.py::test_log_family_s3[Log] test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_grant_and_revoke/test.py::test_allowed_grantees test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used [gw4] [ 1%] PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes [gw4] [ 2%] PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk [gw0] [ 3%] PASSED test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf [gw0] [ 4%] PASSED test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons [gw0] [ 5%] PASSED test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst [gw4] [ 6%] PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin [gw0] [ 7%] PASSED test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp [gw0] [ 8%] PASSED test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump [gw4] [ 9%] PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin [gw0] [ 10%] PASSED test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro [gw0] [ 11%] PASSED test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif [gw1] [ 12%] PASSED test_grant_and_revoke/test.py::test_allowed_grantees [gw0] [ 13%] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_grant_and_revoke/test.py::test_current_database [gw0] [ 14%] PASSED test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld [gw8] [ 15%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] test_log_family_s3/test.py::test_log_family_s3[StripeLog] [gw1] [ 16%] PASSED test_grant_and_revoke/test.py::test_current_database [gw0] [ 17%] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok [gw0] [ 18%] PASSED test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst [gw7] [ 19%] PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart [gw0] [ 20%] PASSED test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_grant_and_revoke/test.py::test_current_grants_override [gw0] [ 21%] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat [gw8] [ 22%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw0] [ 23%] PASSED test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc [gw0] [ 24%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp [gw0] [ 25%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_jbod_balancer/test.py::test_jbod_balanced_merge [gw0] [ 26%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld [gw8] [ 27%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw0] [ 28%] PASSED test_keeper_four_word_command/test.py::test_cmd_ydld [gw1] [ 29%] PASSED test_grant_and_revoke/test.py::test_current_grants_override [gw3] [ 30%] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] test_grant_and_revoke/test.py::test_grant_all_on_table [gw1] [ 31%] PASSED test_grant_and_revoke/test.py::test_grant_all_on_table test_grant_and_revoke/test.py::test_grant_current_grants test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size [gw7] [ 32%] PASSED test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart [gw3] [ 33%] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement [gw3] [ 34%] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement test_insert_over_http_query_log/test.py::test_insert_over_http_ok [gw1] [ 35%] PASSED test_grant_and_revoke/test.py::test_grant_current_grants test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke [gw2] [ 36%] PASSED test_library_bridge/test.py::test_load_all test_library_bridge/test.py::test_load_all_many_rows [gw3] [ 37%] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_ok test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table [gw3] [ 38%] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table [gw1] [ 39%] PASSED test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke test_grant_and_revoke/test.py::test_grant_option [gw1] [ 40%] PASSED test_grant_and_revoke/test.py::test_grant_option test_failed_async_inserts/test.py::test_failed_async_inserts test_grant_and_revoke/test.py::test_grant_with_replace_option [gw2] [ 41%] PASSED test_library_bridge/test.py::test_load_all_many_rows test_library_bridge/test.py::test_load_ids [gw7] [ 42%] PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart [gw2] [ 43%] PASSED test_library_bridge/test.py::test_load_ids test_library_bridge/test.py::test_load_keys test_keeper_session/test.py::test_session_close_shutdown [gw1] [ 44%] PASSED test_grant_and_revoke/test.py::test_grant_with_replace_option test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant [gw2] [ 45%] PASSED test_library_bridge/test.py::test_load_keys test_library_bridge/test.py::test_null_values [gw1] [ 46%] PASSED test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant [gw2] [ 47%] PASSED test_library_bridge/test.py::test_null_values test_library_bridge/test.py::test_path_validation test_grant_and_revoke/test.py::test_implicit_create_view_grant [gw2] [ 48%] PASSED test_library_bridge/test.py::test_path_validation test_library_bridge/test.py::test_recover_after_bridge_crash [gw1] [ 49%] PASSED test_grant_and_revoke/test.py::test_implicit_create_view_grant test_grant_and_revoke/test.py::test_implicit_show_grants test_keeper_internal_secure/test.py::test_secure_raft_works [gw1] [ 50%] PASSED test_grant_and_revoke/test.py::test_implicit_show_grants test_grant_and_revoke/test.py::test_introspection [gw1] [ 51%] PASSED test_grant_and_revoke/test.py::test_introspection test_grant_and_revoke/test.py::test_revoke_requires_grant_option [gw2] [ 52%] PASSED test_library_bridge/test.py::test_recover_after_bridge_crash test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive [gw5] [ 53%] PASSED test_lost_part/test.py::test_lost_last_part test_lost_part/test.py::test_lost_part_mutation [gw6] [ 54%] PASSED test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect [gw1] [ 55%] PASSED test_grant_and_revoke/test.py::test_revoke_requires_grant_option [gw6] [ 56%] PASSED test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart test_grant_and_revoke/test.py::test_smoke [gw1] [ 57%] PASSED test_grant_and_revoke/test.py::test_smoke test_group_array_element_size/test.py::test_max_exement_size [gw6] [ 58%] PASSED test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart [gw9] [ 59%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_http_native/test.py::test_http_native_returns_timezone [gw3] [ 60%] PASSED test_failed_async_inserts/test.py::test_failed_async_inserts [gw0] [ 61%] PASSED test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit test_fetch_memory_usage/test.py::test_huge_column [gw7] [ 62%] PASSED test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout [gw7] [ 63%] PASSED test_keeper_session/test.py::test_session_timeout [gw9] [ 64%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error [gw9] [ 65%] SKIPPED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error test_fix_metadata_version/test.py::test_fix_metadata_version [gw0] [ 66%] PASSED test_memory_limit_observer/test.py::test_observe_memory_limit test_keeper_availability_zone/test.py::test_get_availability_zone [gw4] [ 67%] PASSED test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper [gw8] [ 68%] PASSED test_keeper_internal_secure/test.py::test_secure_raft_works test_keeper_internal_secure/test.py::test_secure_raft_works_with_password [gw5] [ 69%] PASSED test_lost_part/test.py::test_lost_part_mutation test_lost_part/test.py::test_lost_part_other_replica [gw6] [ 70%] PASSED test_http_native/test.py::test_http_native_returns_timezone [gw1] [ 71%] PASSED test_group_array_element_size/test.py::test_max_exement_size test_jbod_ha/test.py::test_jbod_ha [gw2] [ 72%] PASSED test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive test_library_bridge/test.py::test_ssrf [gw2] [ 73%] SKIPPED test_library_bridge/test.py::test_ssrf test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 [gw3] [ 74%] SKIPPED test_fetch_memory_usage/test.py::test_huge_column test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts [gw4] [ 75%] PASSED test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_drop_if_empty/test.py::test_drop_if_empty [gw9] [ 76%] PASSED test_keeper_availability_zone/test.py::test_get_availability_zone test_keeper_mntr_pressure/test.py::test_aggressive_mntr [gw0] [ 77%] PASSED test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper [gw7] [ 78%] PASSED test_fix_metadata_version/test.py::test_fix_metadata_version test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart [gw6] [ 79%] PASSED test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local [gw3] [ 80%] PASSED test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts test_keeper_force_recovery/test.py::test_cluster_recovery [gw8] [ 81%] PASSED test_keeper_internal_secure/test.py::test_secure_raft_works_with_password test_freeze_table/test.py::test_freeze_table test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw4] [ 82%] PASSED test_drop_if_empty/test.py::test_drop_if_empty test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 [gw1] [ 83%] PASSED test_jbod_ha/test.py::test_jbod_ha [gw7] [ 84%] PASSED test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_nodes_move/test.py::test_node_move [gw8] [ 85%] PASSED test_freeze_table/test.py::test_freeze_table test_keeper_secure_client/test.py::test_connection [gw3] [ 86%] PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw0] [ 87%] PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw9] [ 88%] PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_s3_snapshot/test.py::test_s3_upload [gw1] [ 89%] PASSED test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_reconfig_add/test.py::test_reconfig_add test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view [gw4] [ 90%] PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 [gw8] [ 91%] PASSED test_keeper_secure_client/test.py::test_connection [gw7] [ 92%] PASSED test_keeper_nodes_move/test.py::test_node_move [gw2] [ 93%] PASSED test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 [gw1] [ 94%] PASSED test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view [gw9] [ 95%] PASSED test_keeper_reconfig_add/test.py::test_reconfig_add [gw0] [ 96%] PASSED test_keeper_s3_snapshot/test.py::test_s3_upload [gw6] [ 97%] PASSED test_keeper_force_recovery/test.py::test_cluster_recovery [gw6] [ 97%] ERROR test_keeper_force_recovery/test.py::test_cluster_recovery [gw5] [ 98%] PASSED test_lost_part/test.py::test_lost_part_other_replica test_lost_part/test.py::test_lost_part_same_replica [gw5] [ 99%] PASSED test_lost_part/test.py::test_lost_part_same_replica test_library_bridge/test_exiled.py::test_bridge_dies_with_parent [gw5] [100%] SKIPPED test_library_bridge/test_exiled.py::test_bridge_dies_with_parent ==================================== ERRORS ==================================== __________________ ERROR at teardown of test_cluster_recovery __________________ [gw6] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster.start() yield cluster finally: > cluster.shutdown() test_keeper_force_recovery/test.py:48: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , kill = True ignore_fatal = True def shutdown(self, kill=True, ignore_fatal=True): sanitizer_assert_instance = None fatal_log = None if self.up_called: with open(self.docker_logs_path, "w+") as f: try: subprocess.check_call( # STYLE_CHECK_ALLOW_SUBPROCESS_CHECK_CALL self.base_cmd + ["logs"], stdout=f ) except Exception as e: logging.debug("Unable to get logs from docker.") f.seek(0) for line in f: if SANITIZER_SIGN in line: sanitizer_assert_instance = line.split("|")[0].strip() break if kill: try: run_and_check(self.base_cmd + ["stop", "--timeout", "20"]) except Exception as e: logging.debug( "Kill command failed during shutdown. {}".format(repr(e)) ) logging.debug("Trying to kill forcefully") run_and_check(self.base_cmd + ["kill"]) # Check server logs for Fatal messages and sanitizer failures. # NOTE: we cannot do this via docker since in case of Fatal message container may already die. for name, instance in self.instances.items(): if instance.contains_in_log( SANITIZER_SIGN, from_host=True, filename="stderr.log" ): sanitizer_assert_instance = instance.grep_in_log( SANITIZER_SIGN, from_host=True, filename="stderr.log", after=1000, ) logging.error( "Sanitizer in instance %s log %s", name, sanitizer_assert_instance, ) if not ignore_fatal and instance.contains_in_log( "Fatal", from_host=True ): fatal_log = instance.grep_in_log("Fatal", from_host=True) if "Child process was terminated by signal 9 (KILL)" in fatal_log: fatal_log = None continue logging.error("Crash in instance %s fatal log %s", name, fatal_log) try: subprocess_check_call(self.base_cmd + ["down", "--volumes"]) except Exception as e: logging.debug( "Down + remove orphans failed during shutdown. {}".format(repr(e)) ) else: logging.warning( "docker-compose up was not called. Trying to export docker.log for running containers" ) self.cleanup() self.is_up = False self.docker_client = None for instance in list(self.instances.values()): instance.docker_client = None instance.ip_address = None instance.client = None if sanitizer_assert_instance is not None: > raise Exception( "Sanitizer assert found for instance {}".format( sanitizer_assert_instance ) ) E Exception: Sanitizer assert found for instance ================================================================= E ==816==ERROR: AddressSanitizer: heap-use-after-free on address 0x519000099380 at pc 0x55cbb1164d1a bp 0x7ffde1385f80 sp 0x7ffde1385f78 E WRITE of size 1 at 0x519000099380 thread T0 E #0 0x55cbb1164d19 in DB::SnapshotableHashTable::disableSnapshotMode() build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 E #1 0x55cbb1164d19 in DB::KeeperStorage::disableSnapshotMode() build_docker/./src/Coordination/KeeperStorage.h:536:44 E #2 0x55cbb1164d19 in DB::KeeperStorageSnapshot::~KeeperStorageSnapshot() build_docker/./src/Coordination/KeeperSnapshotManager.cpp:563:14 E #3 0x55cbb103c8ca in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 E #4 0x55cbb103c8ca in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 E #5 0x55cbb103c8ca in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 E #6 0x55cbb103c8ca in DB::CreateSnapshotTask::~CreateSnapshotTask() build_docker/./src/Coordination/KeeperSnapshotManager.h:179:8 E #7 0x55cbb103c8ca in void std::__1::__destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 E #8 0x55cbb103c8ca in void std::__1::destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 E #9 0x55cbb103c8ca in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 E #10 0x55cbb103c8ca in std::__1::__deque_base>::clear() build_docker/./contrib/llvm-project/libcxx/include/deque:1261:9 E #11 0x55cbb10225ae in std::__1::__deque_base>::~__deque_base() build_docker/./contrib/llvm-project/libcxx/include/deque:1198:5 E #12 0x55cbb10225ae in ConcurrentBoundedQueue::~ConcurrentBoundedQueue() build_docker/./src/Common/ConcurrentBoundedQueue.h:18:7 E #13 0x55cbb10225ae in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 E #14 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 E #15 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 E #16 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 E #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 E #18 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 E #19 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 E #20 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 E #21 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 E #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 E #23 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 E #24 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 E #25 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 E #26 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 E #27 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 E #28 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 E #29 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 E #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 E #31 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 E #32 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #33 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #34 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 E #36 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 E #37 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #38 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #39 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #40 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 E #41 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 E #42 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 E #43 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 E #44 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 E #45 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 E #46 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 E #47 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 E #48 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 E #49 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 E #50 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 E #51 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 E #52 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 E #53 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) E #54 0x7f85f0703e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) E #55 0x55cb87d418ad in _start (/usr/bin/clickhouse+0xa7208ad) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) E E 0x519000099380 is located 256 bytes inside of 1088-byte region [0x519000099280,0x5190000996c0) E freed by thread T0 here: E #0 0x55cb87e11177 in operator delete(void*, unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7f0177) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) E #1 0x55cbb1215d2e in std::__1::default_delete::operator()[abi:v15000](DB::KeeperStorage*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #2 0x55cbb1215d2e in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperStorage*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #3 0x55cbb1215d2e in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #4 0x55cbb1215d2e in DB::KeeperStateMachine::~KeeperStateMachine() build_docker/./src/Coordination/KeeperStateMachine.h:20:7 E #5 0x55cbb104239d in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 E #6 0x55cbb104239d in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 E #7 0x55cbb104239d in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 E #8 0x55cbb104239d in DB::KeeperServer::~KeeperServer() build_docker/./src/Coordination/KeeperServer.h:20:7 E #9 0x55cbb1021f80 in std::__1::default_delete::operator()[abi:v15000](DB::KeeperServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #10 0x55cbb1021f80 in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #11 0x55cbb1021f80 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #12 0x55cbb1021f80 in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 E #13 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 E #14 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 E #15 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 E #16 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 E #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 E #18 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 E #19 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 E #20 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 E #21 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 E #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 E #23 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 E #24 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 E #25 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 E #26 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 E #27 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 E #28 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 E #29 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 E #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 E #31 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #32 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #33 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #34 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 E #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 E #36 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 E #37 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 E #38 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 E #39 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 E #40 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 E #41 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 E #42 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 E #43 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 E #44 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 E #45 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 E #46 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 E #47 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 E #48 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 E #49 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 E #50 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 E #51 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 E #52 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) E E previously allocated by thread T0 here: E #0 0x55cb87e10332 in operator new(unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7ef332) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) E #1 0x55cbb11760c0 in std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator> const&, std::__1::shared_ptr const&, bool>(unsigned long const&, std::__1::basic_string, std::__1::allocator> const&, std::__1::shared_ptr const&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:28 E #2 0x55cbb11760c0 in DB::KeeperSnapshotManager::deserializeSnapshotFromBuffer(std::__1::shared_ptr) const build_docker/./src/Coordination/KeeperSnapshotManager.cpp:752:22 E #3 0x55cbb11eafc4 in DB::KeeperStateMachine::init() build_docker/./src/Coordination/KeeperStateMachine.cpp:91:69 E #4 0x55cbb106e51c in DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:537:20 E #5 0x55cbb1016813 in DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 E #6 0x55cba97fa016 in DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 E #7 0x55cb9a60c51c in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1609:25 E #8 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 E #9 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 E #10 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 E #11 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 E #12 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 E #13 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) E E SUMMARY: AddressSanitizer: heap-use-after-free build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 in DB::SnapshotableHashTable::disableSnapshotMode() E Shadow bytes around the buggy address: E 0x519000099100: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa E 0x519000099180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa E 0x519000099200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa E 0x519000099280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E =>0x519000099380:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E 0x519000099600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd E Shadow byte legend (one shadow byte represents 8 application bytes): E Addressable: 00 E Partially addressable: 01 02 03 04 05 06 07 E Heap left redzone: fa E Freed heap region: fd E Stack left redzone: f1 E Stack mid redzone: f2 E Stack right redzone: f3 E Stack after return: f5 E Stack use after scope: f8 E Global redzone: f9 E Global init order: f6 E Poisoned by user: f7 E Container overflow: fc E Array cookie: ac E Intra object redzone: bb E ASan internal: fe E Left alloca redzone: ca E Right alloca redzone: cb E ==816==ABORTING E Logging test to /var/log/clickhouse-server/clickhouse-server.log E Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log helpers/cluster.py:3114: Exception ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2025-06-05 19:53:05 [ 527 ] INFO : Running tests in /ClickHouse/tests/integration/test_keeper_force_recovery/test.py (cluster.py:2659, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Cluster start called. is_up=False (cluster.py:2666, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker networks for project roottestkeeperforcerecovery are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker containers for project roottestkeeperforcerecovery are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker volumes for project roottestkeeperforcerecovery are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Cleanup called (cluster.py:801, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker networks for project roottestkeeperforcerecovery are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker containers for project roottestkeeperforcerecovery are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Docker volumes for project roottestkeeperforcerecovery are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2025-06-05 19:53:05 [ 527 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperforcerecovery_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2025-06-05 19:53:05 [ 527 ] DEBUG : Unstopped containers: {} (cluster.py:815, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : No running containers for project: roottestkeeperforcerecovery (cluster.py:829, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Trying to prune unused networks... (cluster.py:835, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Trying to prune unused images... (cluster.py:851, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2025-06-05 19:53:05 [ 527 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2025-06-05 19:53:05 [ 527 ] DEBUG : Images pruned (cluster.py:854, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Trying to prune unused volumes... (cluster.py:860, cleanup) 2025-06-05 19:53:05 [ 527 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2025-06-05 19:53:05 [ 527 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node1 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery/configs/enable_keeper1.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node2 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery/configs/enable_keeper2.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node3 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery/configs/enable_keeper3.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node4 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery/configs/enable_keeper4.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node5 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_force_recovery/configs/enable_keeper5.xml'] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node6 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node7 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup directory for instance: node8 (cluster.py:2679, start) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4383, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Create directory for common tests configuration (cluster.py:4388, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy common configuration from helpers (cluster.py:4408, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Generate and write macros file (cluster.py:4441, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/configs/config.d (cluster.py:4471, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/database (cluster.py:4488, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/logs (cluster.py:4499, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4582, create_dir) 2025-06-05 19:53:05 [ 527 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env (cluster.py:86, _create_env_file) 2025-06-05 19:53:05 [ 527 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-06-05 19:53:05 [ 527 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-06-05 19:53:05 [ 527 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-06-05 19:53:05 [ 527 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-06-05 19:53:05 [ 527 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2025-06-05 19:53:05 [ 527 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env', '--project-name', 'roottestkeeperforcerecovery', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node2 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node6 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node5 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node3 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node4 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node7 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node8 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node5 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node7 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node5 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node5 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node2 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node5 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node2 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node7 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node7 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node3 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node7 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node3 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node1 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node4 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node8 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node6 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node4 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node4 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node4 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node8 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node8 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node8 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node6 ... digest: sha256:8a2c68e2d63d82c826... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node6 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : Stderr:Pulling node6 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:20 [ 527 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env --project-name roottestkeeperforcerecovery --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/docker-compose.yml up -d --no-recreate') (cluster.py:3002, start) 2025-06-05 19:54:20 [ 527 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env', '--project-name', 'roottestkeeperforcerecovery', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating network "roottestkeeperforcerecovery_default" with the default driver (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node7_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node5_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node2_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node4_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node8_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node1_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node6_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node3_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node7_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node5_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node8_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node2_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node4_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node6_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node1_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : Stderr:Creating roottestkeeperforcerecovery_node3_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:54:22 [ 527 ] DEBUG : ClickHouse instance created (cluster.py:3010, start) 2025-06-05 19:54:22 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.35.8... (cluster.py:3017, start) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:22 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:23 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/ac2fbc049e15879ae812e6247719d29f1f9c7032d3f05c96c470f08a64e78ad5/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node1 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.35.5... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/3014029cda2c541efcb0aaa99c7f1f28ed85ced45e901e785139288d53f2e79d/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node2 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.35.9... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/11aa578964977335b08604f860e6e5610dd1595ac8cd44a9dbb386cd019bcfcd/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node3 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node4_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node4, ip: 172.16.35.6... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node4_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/f63e0c378f10e8a9beffcb08dacd2cf2e6333fa197ea02f8d415a8dbedafdfac/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node4 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node5_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node5, ip: 172.16.35.3... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node5_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/1a1b3f48cbfc70a176d393f9d7649b8d4e3b896958130814b750faf6b7800799/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node5 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node6 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node6_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node6, ip: 172.16.35.7... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node6_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/4972fef0eea0b9ab7cfa7442c571b11c21be991222e91f2ec9f28a0fae7d479e/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node6 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node7 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node7_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node7, ip: 172.16.35.2... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node7_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/7a61b0745078dcbb2b588e0d9c52fce9ccac3b2a9929e18c4c232bceccb774cc/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node7 started (cluster.py:3021, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : get_instance_ip instance_name=node8 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node8_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : Waiting for ClickHouse start in node8, ip: 172.16.35.4... (cluster.py:3017, start) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node8_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/c26678e5423a672b4f23e945d11c7e3198f013aa952cf4ad34a6bf2bbb9b6601/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:24 [ 527 ] DEBUG : ClickHouse node8 started (cluster.py:3021, start) ------------------------------ Captured log call ------------------------------- 2025-06-05 19:54:24 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:24 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:24 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:24 [ 527 ] DEBUG : Stdout: 8 ? 00:00:02 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:24 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:24 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:25 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:25 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:25 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:26 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:26 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:26 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:27 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:27 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : Stdout: 8 ? 00:00:02 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:27 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:27 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:27 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:28 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:28 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:28 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:29 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:29 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : Stdout: 8 ? 00:00:02 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:29 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:29 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:29 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:30 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:30 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:31 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:31 [ 527 ] DEBUG : Stdout:700 (cluster.py:121, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:32 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node4_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node5_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] INFO : Connecting to 172.16.35.9(172.16.35.9):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node4_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] INFO : Connecting to 172.16.35.6(172.16.35.6):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:32 [ 527 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:32 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node5_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:32 [ 527 ] INFO : Connecting to 172.16.35.3(172.16.35.3):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node1', data=b'somedata1', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_node1' (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node2', data=b'somedata2', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_node2' (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node3', data=b'somedata3', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_node3' (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node4', data=b'somedata4', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_node4' (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_node5', data=b'somedata5', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_node5' (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=9): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=10): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=11): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=11): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=9): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=10): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=11): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=11): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=9): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=10): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=11): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=11): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=9): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=10): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=11): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=11): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=2): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=2): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=3): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=3): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=4): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=4): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=5): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=5): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=6): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=6): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=7): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=7): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=8): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=8): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=9): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=9): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=10): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=10): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : Sending request(xid=11): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:32 [ 527 ] DEBUG : Received response(xid=11): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:54:32 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:32 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : Stdout: 8 ? 00:00:02 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:32 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:32 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:32 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:33 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:33 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:33 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:34 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:34 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:34 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:35 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:35 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:35 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:36 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:36 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:36 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:37 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:37 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:37 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:38 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:54:38 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:38 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:38 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:38 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:54:38 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:38 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:54:38 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:38 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:39 [ 527 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:39 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:39 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=1): Create(path='/test_force_recovery_extra', data=b'somedataextra', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=1): '/test_force_recovery_extra' (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=12): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=12): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=13): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=13): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=12): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=12): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=13): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=13): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=12): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=12): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : Sending request(xid=13): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:39 [ 527 ] DEBUG : Received response(xid=13): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:54:39 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:39 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:39 [ 527 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3817, start_clickhouse) 2025-06-05 19:54:39 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:39 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2025-06-05 19:54:39 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:39 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:54:40 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:40 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:54:40 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:40 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:40 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:54:40 [ 527 ] DEBUG : Clickhouse process running. (cluster.py:3828, start_clickhouse) 2025-06-05 19:54:40 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:40 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:40 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:54:40 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:54:40 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:54:41 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:54:42 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:54:42 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:42 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:42 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:42 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:42 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] DEBUG : Received response(xid=2): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=14): Close() (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:42 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=5, time_out=30000, session_id=1, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:54:42 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=14): Close() (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:42 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:54:42 [ 527 ] DEBUG : Sending request(xid=14): Close() (connection.py:312, _submit) 2025-06-05 19:54:42 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:42 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:54:42 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:54:42 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node3_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:42 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node3_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:42 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:42 [ 527 ] DEBUG : Stdout: 8 ? 00:00:03 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:42 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node3_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:42 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node3_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:42 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:42 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:42 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:43 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:43 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:43 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node3_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:44 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node3_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node4_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:44 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node4_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : Stdout: 8 ? 00:00:03 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node4_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:44 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node4_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:44 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node4_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:44 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node4_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:45 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:46 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node4_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:46 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node4_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:46 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node4_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:47 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node4_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:47 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : Stdout: 8 ? 00:00:03 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:47 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:47 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:47 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:48 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:48 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:48 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:49 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:49 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:49 [ 527 ] DEBUG : Stdout:8 (cluster.py:121, run_and_check) 2025-06-05 19:54:50 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node5_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:54:50 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node5_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:50 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:50 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:51 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:51 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:52 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:52 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:53 [ 527 ] WARNING : Connection dropped: outstanding heartbeat ping not received (connection.py:622, _connect_attempt) 2025-06-05 19:54:53 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:53 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:53 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:53 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:54 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=5, time_out=30000, session_id=2, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] WARNING : Connection dropped: outstanding heartbeat ping not received (connection.py:622, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:54 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=16, time_out=30000, session_id=6, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] WARNING : Connection dropped: outstanding heartbeat ping not received (connection.py:622, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:54 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=21, time_out=30000, session_id=7, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] WARNING : Connection dropped: outstanding heartbeat ping not received (connection.py:622, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:54:54 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:54 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=5, time_out=30000, session_id=8, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:54:54 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:54 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:55 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:55 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:56 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:56 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:57 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:57 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:58 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:58 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:54:59 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:54:59 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:00 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:00 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper1.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjE8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper1.xml'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:00 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper1.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjE8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper1.xml'] (cluster.py:113, run_and_check) 2025-06-05 19:55:00 [ 527 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node1 (cluster.py:3455, query) 2025-06-05 19:55:00 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:00 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:00 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:00 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:00 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper6.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjY8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper6.xml'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:00 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper6.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjY8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper6.xml'] (cluster.py:113, run_and_check) 2025-06-05 19:55:00 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:00 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:00 [ 527 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3817, start_clickhouse) 2025-06-05 19:55:00 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:00 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2025-06-05 19:55:01 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:01 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:01 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:01 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:01 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:01 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:01 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:01 [ 527 ] DEBUG : Clickhouse process running. (cluster.py:3828, start_clickhouse) 2025-06-05 19:55:01 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node6_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:01 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node6_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:02 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:02 [ 527 ] DEBUG : Executing query select 20 on node6 (cluster.py:3455, query) 2025-06-05 19:55:02 [ 527 ] DEBUG : Executing query select 20 on node6 (cluster.py:3455, query) 2025-06-05 19:55:03 [ 527 ] DEBUG : Executing query select 20 on node6 (cluster.py:3455, query) 2025-06-05 19:55:03 [ 527 ] DEBUG : get_instance_ip instance_name=node6 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:03 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node6_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:03 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:03 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:03 [ 527 ] DEBUG : get_instance_ip instance_name=node6 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:03 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node6_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:03 [ 527 ] INFO : Connecting to 172.16.35.7(172.16.35.7):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:03 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:03 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:03 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper7.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjc8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper7.xml'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:03 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper7.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjc8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper7.xml'] (cluster.py:113, run_and_check) 2025-06-05 19:55:03 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:03 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:03 [ 527 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3817, start_clickhouse) 2025-06-05 19:55:03 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:03 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2025-06-05 19:55:04 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:04 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:05 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:05 [ 527 ] DEBUG : Clickhouse process running. (cluster.py:3828, start_clickhouse) 2025-06-05 19:55:05 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node7_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:05 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node7_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:05 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:05 [ 527 ] DEBUG : Executing query select 20 on node7 (cluster.py:3455, query) 2025-06-05 19:55:05 [ 527 ] DEBUG : Executing query select 20 on node7 (cluster.py:3455, query) 2025-06-05 19:55:06 [ 527 ] DEBUG : Executing query select 20 on node7 (cluster.py:3455, query) 2025-06-05 19:55:07 [ 527 ] DEBUG : Executing query select 20 on node7 (cluster.py:3455, query) 2025-06-05 19:55:07 [ 527 ] DEBUG : get_instance_ip instance_name=node7 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:07 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node7_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:07 [ 527 ] DEBUG : get_instance_ip instance_name=node7 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:07 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node7_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:07 [ 527 ] INFO : Connecting to 172.16.35.2(172.16.35.2):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:07 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:07 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:07 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper8.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjg8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper8.xml'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:07 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/enable_keeper8.xml) && echo PGNsaWNraG91c2U+CiAgICA8a2VlcGVyX3NlcnZlcj4KICAgICAgICA8dXNlX2NsdXN0ZXI+ZmFsc2U8L3VzZV9jbHVzdGVyPgogICAgICAgIDx0Y3BfcG9ydD45MTgxPC90Y3BfcG9ydD4KICAgICAgICA8c2VydmVyX2lkPjg8L3NlcnZlcl9pZD4KICAgICAgICA8bG9nX3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9sb2c8L2xvZ19zdG9yYWdlX3BhdGg+CiAgICAgICAgPHNuYXBzaG90X3N0b3JhZ2VfcGF0aD4vdmFyL2xpYi9jbGlja2hvdXNlL2Nvb3JkaW5hdGlvbi9zbmFwc2hvdHM8L3NuYXBzaG90X3N0b3JhZ2VfcGF0aD4KCiAgICAgICAgPGNvb3JkaW5hdGlvbl9zZXR0aW5ncz4KICAgICAgICAgICAgPG9wZXJhdGlvbl90aW1lb3V0X21zPjUwMDA8L29wZXJhdGlvbl90aW1lb3V0X21zPgogICAgICAgICAgICA8c2Vzc2lvbl90aW1lb3V0X21zPjEwMDAwPC9zZXNzaW9uX3RpbWVvdXRfbXM+CiAgICAgICAgICAgIDxzbmFwc2hvdF9kaXN0YW5jZT43NTwvc25hcHNob3RfZGlzdGFuY2U+CiAgICAgICAgICAgIDxyYWZ0X2xvZ3NfbGV2ZWw+dHJhY2U8L3JhZnRfbG9nc19sZXZlbD4KICAgICAgICAgICAgPG1heF9yZXF1ZXN0c19iYXRjaF9zaXplPjIwMDwvbWF4X3JlcXVlc3RzX2JhdGNoX3NpemU+CiAgICAgICAgPC9jb29yZGluYXRpb25fc2V0dGluZ3M+CgogICAgICAgIDxyYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MTwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTE8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+MjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTI8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NjwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTY8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+NzwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTc8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgICAgIDxzZXJ2ZXI+CiAgICAgICAgICAgICAgICA8aWQ+ODwvaWQ+CiAgICAgICAgICAgICAgICA8aG9zdG5hbWU+bm9kZTg8L2hvc3RuYW1lPgogICAgICAgICAgICAgICAgPHBvcnQ+OTIzNDwvcG9ydD4KICAgICAgICAgICAgPC9zZXJ2ZXI+CiAgICAgICAgPC9yYWZ0X2NvbmZpZ3VyYXRpb24+CiAgICA8L2tlZXBlcl9zZXJ2ZXI+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/enable_keeper8.xml'] (cluster.py:113, run_and_check) 2025-06-05 19:55:07 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:07 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:07 [ 527 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3817, start_clickhouse) 2025-06-05 19:55:07 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:07 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2025-06-05 19:55:08 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:08 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:08 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:08 [ 527 ] DEBUG : Clickhouse process running. (cluster.py:3828, start_clickhouse) 2025-06-05 19:55:08 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node8_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:08 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node8_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:08 [ 527 ] DEBUG : Stdout:747 (cluster.py:121, run_and_check) 2025-06-05 19:55:08 [ 527 ] DEBUG : Executing query select 20 on node8 (cluster.py:3455, query) 2025-06-05 19:55:09 [ 527 ] DEBUG : Executing query select 20 on node8 (cluster.py:3455, query) 2025-06-05 19:55:10 [ 527 ] DEBUG : Executing query select 20 on node8 (cluster.py:3455, query) 2025-06-05 19:55:10 [ 527 ] DEBUG : Executing query select 20 on node8 (cluster.py:3455, query) 2025-06-05 19:55:10 [ 527 ] DEBUG : get_instance_ip instance_name=node8 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:10 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node8_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:10 [ 527 ] DEBUG : get_instance_ip instance_name=node8 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:10 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node8_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:10 [ 527 ] INFO : Connecting to 172.16.35.4(172.16.35.4):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:10 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:10 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:10 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:10 [ 527 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:10 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:10 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:10 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:10 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:11 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:11 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:11 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:55:11 [ 527 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2025-06-05 19:55:11 [ 527 ] DEBUG : Stdout: 816 ? 00:00:04 clickhouse (cluster.py:121, run_and_check) 2025-06-05 19:55:11 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:11 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2025-06-05 19:55:11 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:11 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:11 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:12 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:12 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:12 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:13 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:13 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:13 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:14 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:14 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:14 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:15 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:15 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:15 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:16 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:16 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:16 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:17 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:17 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:17 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:17 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:18 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:18 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:18 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:18 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:18 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:18 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:18 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:18 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:19 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:19 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:19 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:19 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:19 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:19 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:19 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:20 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:20 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:20 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:20 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:20 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:20 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:21 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:22 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:22 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:22 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:22 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:22 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:22 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:22 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:23 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:23 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:23 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:23 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:23 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:24 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:24 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:24 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:24 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:24 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:25 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:25 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:25 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:26 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:26 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:26 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:27 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:27 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:27 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:27 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:27 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:28 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:28 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:28 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:28 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:28 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:28 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:28 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:29 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:29 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:29 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:30 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:30 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:30 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:31 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:31 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:31 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:32 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:32 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:32 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:33 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:33 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:33 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:33 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:34 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:35 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:35 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:35 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:36 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:36 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:36 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:37 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:37 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:37 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:38 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:38 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:38 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:39 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:39 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:39 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:39 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:39 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:40 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:40 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:40 [ 527 ] DEBUG : Stdout:816 (cluster.py:121, run_and_check) 2025-06-05 19:55:40 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:40 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:41 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:41 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps aux'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:41 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'ps aux'] (cluster.py:113, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : Stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND (cluster.py:121, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : Stdout:root 1 0.0 0.0 7372 3692 ? Ss 19:54 0:00 bash -c trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $! (cluster.py:121, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : Stdout:root 10 0.0 0.0 5804 992 ? S 19:54 0:00 tail -f /dev/null (cluster.py:121, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : Stdout:root 1906 0.0 0.0 10072 1580 ? Rs 19:55 0:00 ps aux (cluster.py:121, run_and_check) 2025-06-05 19:55:41 [ 527 ] WARNING : We want force stop clickhouse, but no clickhouse-server is running USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 7372 3692 ? Ss 19:54 0:00 bash -c trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $! root 10 0.0 0.0 5804 992 ? S 19:54 0:00 tail -f /dev/null root 1906 0.0 0.0 10072 1580 ? Rs 19:55 0:00 ps aux (cluster.py:3795, stop_clickhouse) 2025-06-05 19:55:41 [ 527 ] DEBUG : Sending request(xid=13): Create(path='/test_force_recovery_last', data=b'somedatalast', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) (connection.py:312, _submit) 2025-06-05 19:55:41 [ 527 ] DEBUG : Received response(xid=13): '/test_force_recovery_last' (connection.py:410, _read_response) 2025-06-05 19:55:41 [ 527 ] DEBUG : Sending request(xid=13): Exists(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:41 [ 527 ] DEBUG : Received response(xid=13): ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119) (connection.py:410, _read_response) 2025-06-05 19:55:41 [ 527 ] DEBUG : Sending request(xid=14): GetData(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:41 [ 527 ] DEBUG : Received response(xid=14): (b'somedatalast', ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119)) (connection.py:410, _read_response) 2025-06-05 19:55:41 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:41 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:41 [ 527 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3817, start_clickhouse) 2025-06-05 19:55:41 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:41 [ 527 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2025-06-05 19:55:42 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:42 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:43 [ 527 ] DEBUG : Stdout:1930 (cluster.py:121, run_and_check) 2025-06-05 19:55:43 [ 527 ] DEBUG : Clickhouse process running. (cluster.py:3828, start_clickhouse) 2025-06-05 19:55:43 [ 527 ] DEBUG : run container_id:roottestkeeperforcerecovery_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2046, exec_in_container) 2025-06-05 19:55:43 [ 527 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperforcerecovery_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2025-06-05 19:55:43 [ 527 ] DEBUG : Stdout:1930 (cluster.py:121, run_and_check) 2025-06-05 19:55:43 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:55:43 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:55:44 [ 527 ] DEBUG : Executing query select 20 on node1 (cluster.py:3455, query) 2025-06-05 19:55:44 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:44 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:44 [ 527 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2008, get_instance_ip) 2025-06-05 19:55:44 [ 527 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperforcerecovery_node1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2025-06-05 19:55:44 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=1): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=1): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=2): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=2): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=3): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=3): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=4): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=4): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=5): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=5): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=6): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=6): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=7): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=7): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=8): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=8): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=9): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=9): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=10): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=10): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=11): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=11): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=12): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=12): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=13): Exists(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=13): ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=14): GetData(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=14): (b'somedatalast', ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=13): Exists(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=13): ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=14): GetData(path='/test_force_recovery_node1', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=14): (b'somedata1', ZnodeStat(czxid=1, mzxid=1, ctime=1749153272206, mtime=1749153272206, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=1)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=15): Exists(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=15): ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=16): GetData(path='/test_force_recovery_node2', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=16): (b'somedata2', ZnodeStat(czxid=2, mzxid=2, ctime=1749153272211, mtime=1749153272211, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=2)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=17): Exists(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=17): ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=18): GetData(path='/test_force_recovery_node3', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=18): (b'somedata3', ZnodeStat(czxid=3, mzxid=3, ctime=1749153272218, mtime=1749153272218, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=3)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=19): Exists(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=19): ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=20): GetData(path='/test_force_recovery_node4', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=20): (b'somedata4', ZnodeStat(czxid=4, mzxid=4, ctime=1749153272224, mtime=1749153272224, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=4)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=21): Exists(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=21): ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=22): GetData(path='/test_force_recovery_node5', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=22): (b'somedata5', ZnodeStat(czxid=5, mzxid=5, ctime=1749153272232, mtime=1749153272232, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=9, numChildren=0, pzxid=5)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=23): Exists(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=23): ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=24): GetData(path='/test_force_recovery_extra', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=24): (b'somedataextra', ZnodeStat(czxid=16, mzxid=16, ctime=1749153279028, mtime=1749153279028, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=16)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=25): Exists(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=25): ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=26): GetData(path='/test_force_recovery_last', watcher=None) (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] DEBUG : Received response(xid=26): (b'somedatalast', ZnodeStat(czxid=119, mzxid=119, ctime=1749153341694, mtime=1749153341694, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=119)) (connection.py:410, _read_response) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=15): Close() (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:44 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=27): Close() (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:44 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=13): Close() (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:44 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:55:44 [ 527 ] DEBUG : Sending request(xid=14): Close() (connection.py:312, _submit) 2025-06-05 19:55:44 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:44 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:45 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:55:45 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-06-05 19:55:45 [ 527 ] DEBUG : Sending request(xid=15): Close() (connection.py:312, _submit) 2025-06-05 19:55:45 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:45 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:45 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:45 [ 527 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-06-05 19:55:45 [ 527 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) ---------------------------- Captured log teardown ----------------------------- 2025-06-05 19:55:45 [ 527 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env', '--project-name', 'roottestkeeperforcerecovery', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2025-06-05 19:55:46 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:46 [ 527 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-06-05 19:55:46 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:46 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:46 [ 527 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-06-05 19:55:46 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:46 [ 527 ] INFO : Connecting to 172.16.35.5(172.16.35.5):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node3_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node6_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node1_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node4_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node2_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node5_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node8_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node7_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node3_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node4_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node5_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node8_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node7_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node2_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node1_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Stopping roottestkeeperforcerecovery_node6_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs/stderr.log:================================================================= (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : grep in log called ================== (cluster.py:3946, grep_in_log) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs/stderr.log ] && zgrep -A1000 -a "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:================================================================= (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:==816==ERROR: AddressSanitizer: heap-use-after-free on address 0x519000099380 at pc 0x55cbb1164d1a bp 0x7ffde1385f80 sp 0x7ffde1385f78 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:WRITE of size 1 at 0x519000099380 thread T0 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #0 0x55cbb1164d19 in DB::SnapshotableHashTable::disableSnapshotMode() build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #1 0x55cbb1164d19 in DB::KeeperStorage::disableSnapshotMode() build_docker/./src/Coordination/KeeperStorage.h:536:44 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #2 0x55cbb1164d19 in DB::KeeperStorageSnapshot::~KeeperStorageSnapshot() build_docker/./src/Coordination/KeeperSnapshotManager.cpp:563:14 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #3 0x55cbb103c8ca in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #4 0x55cbb103c8ca in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #5 0x55cbb103c8ca in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #6 0x55cbb103c8ca in DB::CreateSnapshotTask::~CreateSnapshotTask() build_docker/./src/Coordination/KeeperSnapshotManager.h:179:8 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #7 0x55cbb103c8ca in void std::__1::__destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #8 0x55cbb103c8ca in void std::__1::destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #9 0x55cbb103c8ca in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #10 0x55cbb103c8ca in std::__1::__deque_base>::clear() build_docker/./contrib/llvm-project/libcxx/include/deque:1261:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #11 0x55cbb10225ae in std::__1::__deque_base>::~__deque_base() build_docker/./contrib/llvm-project/libcxx/include/deque:1198:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #12 0x55cbb10225ae in ConcurrentBoundedQueue::~ConcurrentBoundedQueue() build_docker/./src/Common/ConcurrentBoundedQueue.h:18:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #13 0x55cbb10225ae in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #14 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #15 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #16 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #18 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #19 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #20 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #21 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #23 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #24 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #25 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #26 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #27 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #28 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #29 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #31 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #32 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #33 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #34 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #36 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #37 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #38 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #39 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #40 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #41 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #42 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #43 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #44 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #45 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #46 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #47 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #48 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #49 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #50 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #51 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #52 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #53 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #54 0x7f85f0703e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #55 0x55cb87d418ad in _start (/usr/bin/clickhouse+0xa7208ad) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:0x519000099380 is located 256 bytes inside of 1088-byte region [0x519000099280,0x5190000996c0) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:freed by thread T0 here: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #0 0x55cb87e11177 in operator delete(void*, unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7f0177) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #1 0x55cbb1215d2e in std::__1::default_delete::operator()[abi:v15000](DB::KeeperStorage*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #2 0x55cbb1215d2e in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperStorage*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #3 0x55cbb1215d2e in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #4 0x55cbb1215d2e in DB::KeeperStateMachine::~KeeperStateMachine() build_docker/./src/Coordination/KeeperStateMachine.h:20:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #5 0x55cbb104239d in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #6 0x55cbb104239d in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #7 0x55cbb104239d in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #8 0x55cbb104239d in DB::KeeperServer::~KeeperServer() build_docker/./src/Coordination/KeeperServer.h:20:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #9 0x55cbb1021f80 in std::__1::default_delete::operator()[abi:v15000](DB::KeeperServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #10 0x55cbb1021f80 in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #11 0x55cbb1021f80 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #12 0x55cbb1021f80 in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #13 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #14 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #15 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #16 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #18 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #19 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #20 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #21 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #23 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #24 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #25 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #26 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #27 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #28 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #29 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #31 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #32 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #33 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #34 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #36 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #37 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #38 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #39 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #40 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #41 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #42 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #43 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #44 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #45 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #46 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #47 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #48 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #49 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #50 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #51 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #52 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:previously allocated by thread T0 here: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #0 0x55cb87e10332 in operator new(unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7ef332) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #1 0x55cbb11760c0 in std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator> const&, std::__1::shared_ptr const&, bool>(unsigned long const&, std::__1::basic_string, std::__1::allocator> const&, std::__1::shared_ptr const&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:28 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #2 0x55cbb11760c0 in DB::KeeperSnapshotManager::deserializeSnapshotFromBuffer(std::__1::shared_ptr) const build_docker/./src/Coordination/KeeperSnapshotManager.cpp:752:22 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #3 0x55cbb11eafc4 in DB::KeeperStateMachine::init() build_docker/./src/Coordination/KeeperStateMachine.cpp:91:69 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #4 0x55cbb106e51c in DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:537:20 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #5 0x55cbb1016813 in DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #6 0x55cba97fa016 in DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #7 0x55cb9a60c51c in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1609:25 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #8 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #9 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #10 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #11 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #12 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: #13 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:SUMMARY: AddressSanitizer: heap-use-after-free build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 in DB::SnapshotableHashTable::disableSnapshotMode() (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:Shadow bytes around the buggy address: (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099100: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:=>0x519000099380:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: 0x519000099600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:Shadow byte legend (one shadow byte represents 8 application bytes): (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Addressable: 00 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Partially addressable: 01 02 03 04 05 06 07 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Heap left redzone: fa (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Freed heap region: fd (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Stack left redzone: f1 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Stack mid redzone: f2 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Stack right redzone: f3 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Stack after return: f5 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Stack use after scope: f8 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Global redzone: f9 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Global init order: f6 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Poisoned by user: f7 (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Container overflow: fc (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Array cookie: ac (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Intra object redzone: bb (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: ASan internal: fe (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Left alloca redzone: ca (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout: Right alloca redzone: cb (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:==816==ABORTING (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:Logging test to /var/log/clickhouse-server/clickhouse-server.log (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : grep result ================================================================= ==816==ERROR: AddressSanitizer: heap-use-after-free on address 0x519000099380 at pc 0x55cbb1164d1a bp 0x7ffde1385f80 sp 0x7ffde1385f78 WRITE of size 1 at 0x519000099380 thread T0 #0 0x55cbb1164d19 in DB::SnapshotableHashTable::disableSnapshotMode() build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 #1 0x55cbb1164d19 in DB::KeeperStorage::disableSnapshotMode() build_docker/./src/Coordination/KeeperStorage.h:536:44 #2 0x55cbb1164d19 in DB::KeeperStorageSnapshot::~KeeperStorageSnapshot() build_docker/./src/Coordination/KeeperSnapshotManager.cpp:563:14 #3 0x55cbb103c8ca in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #4 0x55cbb103c8ca in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #5 0x55cbb103c8ca in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #6 0x55cbb103c8ca in DB::CreateSnapshotTask::~CreateSnapshotTask() build_docker/./src/Coordination/KeeperSnapshotManager.h:179:8 #7 0x55cbb103c8ca in void std::__1::__destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #8 0x55cbb103c8ca in void std::__1::destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #9 0x55cbb103c8ca in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #10 0x55cbb103c8ca in std::__1::__deque_base>::clear() build_docker/./contrib/llvm-project/libcxx/include/deque:1261:9 #11 0x55cbb10225ae in std::__1::__deque_base>::~__deque_base() build_docker/./contrib/llvm-project/libcxx/include/deque:1198:5 #12 0x55cbb10225ae in ConcurrentBoundedQueue::~ConcurrentBoundedQueue() build_docker/./src/Common/ConcurrentBoundedQueue.h:18:7 #13 0x55cbb10225ae in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 #14 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #15 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #16 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #18 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #19 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #20 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #21 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #23 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #24 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #25 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #26 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #27 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 #28 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 #29 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #31 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #32 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #33 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #34 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #36 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #37 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #38 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #39 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #40 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 #41 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #42 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #43 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #44 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 #45 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 #46 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 #47 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 #48 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #49 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #50 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #51 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #52 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #53 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) #54 0x7f85f0703e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) #55 0x55cb87d418ad in _start (/usr/bin/clickhouse+0xa7208ad) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) 0x519000099380 is located 256 bytes inside of 1088-byte region [0x519000099280,0x5190000996c0) freed by thread T0 here: #0 0x55cb87e11177 in operator delete(void*, unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7f0177) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) #1 0x55cbb1215d2e in std::__1::default_delete::operator()[abi:v15000](DB::KeeperStorage*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #2 0x55cbb1215d2e in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperStorage*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #3 0x55cbb1215d2e in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #4 0x55cbb1215d2e in DB::KeeperStateMachine::~KeeperStateMachine() build_docker/./src/Coordination/KeeperStateMachine.h:20:7 #5 0x55cbb104239d in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #6 0x55cbb104239d in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #7 0x55cbb104239d in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #8 0x55cbb104239d in DB::KeeperServer::~KeeperServer() build_docker/./src/Coordination/KeeperServer.h:20:7 #9 0x55cbb1021f80 in std::__1::default_delete::operator()[abi:v15000](DB::KeeperServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #10 0x55cbb1021f80 in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #11 0x55cbb1021f80 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #12 0x55cbb1021f80 in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 #13 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #14 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #15 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #16 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #18 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #19 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #20 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #21 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #23 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #24 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #25 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #26 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 #27 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 #28 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 #29 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #31 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #32 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #33 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #34 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #36 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #37 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #38 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #39 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 #40 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #41 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #42 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #43 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 #44 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 #45 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 #46 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 #47 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #48 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #49 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #50 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #51 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #52 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) previously allocated by thread T0 here: #0 0x55cb87e10332 in operator new(unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7ef332) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) #1 0x55cbb11760c0 in std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator> const&, std::__1::shared_ptr const&, bool>(unsigned long const&, std::__1::basic_string, std::__1::allocator> const&, std::__1::shared_ptr const&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:28 #2 0x55cbb11760c0 in DB::KeeperSnapshotManager::deserializeSnapshotFromBuffer(std::__1::shared_ptr) const build_docker/./src/Coordination/KeeperSnapshotManager.cpp:752:22 #3 0x55cbb11eafc4 in DB::KeeperStateMachine::init() build_docker/./src/Coordination/KeeperStateMachine.cpp:91:69 #4 0x55cbb106e51c in DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:537:20 #5 0x55cbb1016813 in DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 #6 0x55cba97fa016 in DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 #7 0x55cb9a60c51c in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1609:25 #8 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #9 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #10 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #11 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #12 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #13 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) SUMMARY: AddressSanitizer: heap-use-after-free build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 in DB::SnapshotableHashTable::disableSnapshotMode() Shadow bytes around the buggy address: 0x519000099100: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x519000099380:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==816==ABORTING Logging test to /var/log/clickhouse-server/clickhouse-server.log Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:3968, grep_in_log) 2025-06-05 19:55:47 [ 527 ] ERROR : Sanitizer in instance node1 log ================================================================= ==816==ERROR: AddressSanitizer: heap-use-after-free on address 0x519000099380 at pc 0x55cbb1164d1a bp 0x7ffde1385f80 sp 0x7ffde1385f78 WRITE of size 1 at 0x519000099380 thread T0 #0 0x55cbb1164d19 in DB::SnapshotableHashTable::disableSnapshotMode() build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 #1 0x55cbb1164d19 in DB::KeeperStorage::disableSnapshotMode() build_docker/./src/Coordination/KeeperStorage.h:536:44 #2 0x55cbb1164d19 in DB::KeeperStorageSnapshot::~KeeperStorageSnapshot() build_docker/./src/Coordination/KeeperSnapshotManager.cpp:563:14 #3 0x55cbb103c8ca in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #4 0x55cbb103c8ca in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #5 0x55cbb103c8ca in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #6 0x55cbb103c8ca in DB::CreateSnapshotTask::~CreateSnapshotTask() build_docker/./src/Coordination/KeeperSnapshotManager.h:179:8 #7 0x55cbb103c8ca in void std::__1::__destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #8 0x55cbb103c8ca in void std::__1::destroy_at[abi:v15000](DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #9 0x55cbb103c8ca in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::CreateSnapshotTask*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #10 0x55cbb103c8ca in std::__1::__deque_base>::clear() build_docker/./contrib/llvm-project/libcxx/include/deque:1261:9 #11 0x55cbb10225ae in std::__1::__deque_base>::~__deque_base() build_docker/./contrib/llvm-project/libcxx/include/deque:1198:5 #12 0x55cbb10225ae in ConcurrentBoundedQueue::~ConcurrentBoundedQueue() build_docker/./src/Common/ConcurrentBoundedQueue.h:18:7 #13 0x55cbb10225ae in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 #14 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #15 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #16 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #18 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #19 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #20 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #21 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #23 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #24 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #25 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #26 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #27 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 #28 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 #29 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #31 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #32 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #33 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #34 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #36 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #37 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #38 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #39 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #40 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 #41 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #42 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #43 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #44 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 #45 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 #46 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 #47 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 #48 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #49 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #50 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #51 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #52 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #53 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) #54 0x7f85f0703e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) #55 0x55cb87d418ad in _start (/usr/bin/clickhouse+0xa7208ad) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) 0x519000099380 is located 256 bytes inside of 1088-byte region [0x519000099280,0x5190000996c0) freed by thread T0 here: #0 0x55cb87e11177 in operator delete(void*, unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7f0177) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) #1 0x55cbb1215d2e in std::__1::default_delete::operator()[abi:v15000](DB::KeeperStorage*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #2 0x55cbb1215d2e in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperStorage*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #3 0x55cbb1215d2e in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #4 0x55cbb1215d2e in DB::KeeperStateMachine::~KeeperStateMachine() build_docker/./src/Coordination/KeeperStateMachine.h:20:7 #5 0x55cbb104239d in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #6 0x55cbb104239d in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #7 0x55cbb104239d in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #8 0x55cbb104239d in DB::KeeperServer::~KeeperServer() build_docker/./src/Coordination/KeeperServer.h:20:7 #9 0x55cbb1021f80 in std::__1::default_delete::operator()[abi:v15000](DB::KeeperServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #10 0x55cbb1021f80 in std::__1::unique_ptr>::reset[abi:v15000](DB::KeeperServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #11 0x55cbb1021f80 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #12 0x55cbb1021f80 in DB::KeeperDispatcher::~KeeperDispatcher() build_docker/./src/Coordination/KeeperDispatcher.cpp:621:1 #13 0x55cb9a362f0f in std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 #14 0x55cb9a362f0f in std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 #15 0x55cb9a362f0f in std::__1::shared_ptr::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 #16 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #17 0x55cb9a362f0f in DB::KeeperTCPHandlerFactory::~KeeperTCPHandlerFactory() build_docker/./src/Server/KeeperTCPHandlerFactory.h:15:7 #18 0x55cbafda2727 in Poco::ReleasePolicy::release(DB::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #19 0x55cbafda2727 in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #20 0x55cbafda2727 in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #21 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #22 0x55cbafda2727 in DB::TCPServerConnectionFactoryImpl::~TCPServerConnectionFactoryImpl() build_docker/./src/Server/TCPServer.cpp:7:7 #23 0x55cbb787e52f in Poco::ReleasePolicy::release(Poco::Net::TCPServerConnectionFactory*) build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:58:9 #24 0x55cbb787e52f in Poco::SharedPtr>::release() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:296:13 #25 0x55cbb787e52f in Poco::SharedPtr>::~SharedPtr() build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:129:13 #26 0x55cbb787e52f in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:79:1 #27 0x55cbb787e6ad in Poco::Net::TCPServerDispatcher::~TCPServerDispatcher() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:78:1 #28 0x55cbb787d0ef in Poco::Net::TCPServer::~TCPServer() build_docker/./base/poco/Net/src/TCPServer.cpp:91:17 #29 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #30 0x55cbafda246a in DB::TCPServer::~TCPServer() build_docker/./src/Server/TCPServer.h:13:7 #31 0x55cbafd22eba in std::__1::default_delete::operator()[abi:v15000](DB::TCPServer*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #32 0x55cbafd22eba in std::__1::unique_ptr>::reset[abi:v15000](DB::TCPServer*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #33 0x55cbafd22eba in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #34 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #35 0x55cbafd22eba in DB::ProtocolServerAdapter::TCPServerAdapterImpl::~TCPServerAdapterImpl() build_docker/./src/Server/ProtocolServerAdapter.cpp:15:46 #36 0x55cb9a618a39 in std::__1::default_delete::operator()[abi:v15000](DB::ProtocolServerAdapter::Impl*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 #37 0x55cb9a618a39 in std::__1::unique_ptr>::reset[abi:v15000](DB::ProtocolServerAdapter::Impl*) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305:7 #38 0x55cb9a618a39 in std::__1::unique_ptr>::~unique_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259:19 #39 0x55cb9a618a39 in DB::ProtocolServerAdapter::~ProtocolServerAdapter() build_docker/./src/Server/ProtocolServerAdapter.h:18:7 #40 0x55cb9a618a39 in void std::__1::__destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:63:13 #41 0x55cb9a618a39 in void std::__1::destroy_at[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/construct_at.h:88:5 #42 0x55cb9a618a39 in void std::__1::allocator_traits>::destroy[abi:v15000](std::__1::allocator&, DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:317:9 #43 0x55cb9a618a39 in std::__1::vector>::__base_destruct_at_end[abi:v15000](DB::ProtocolServerAdapter*) build_docker/./contrib/llvm-project/libcxx/include/vector:833:9 #44 0x55cb9a618a39 in std::__1::vector>::__clear[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:827:29 #45 0x55cb9a618a39 in std::__1::vector>::~vector[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/vector:436:9 #46 0x55cb9a618a39 in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:2165:1 #47 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #48 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #49 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #50 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #51 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #52 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) previously allocated by thread T0 here: #0 0x55cb87e10332 in operator new(unsigned long, std::align_val_t) (/usr/bin/clickhouse+0xa7ef332) (BuildId: 9bcf28f9ec83b253301fe91c56199b2326089e61) #1 0x55cbb11760c0 in std::__1::__unique_if::__unique_single std::__1::make_unique[abi:v15000], std::__1::allocator> const&, std::__1::shared_ptr const&, bool>(unsigned long const&, std::__1::basic_string, std::__1::allocator> const&, std::__1::shared_ptr const&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:28 #2 0x55cbb11760c0 in DB::KeeperSnapshotManager::deserializeSnapshotFromBuffer(std::__1::shared_ptr) const build_docker/./src/Coordination/KeeperSnapshotManager.cpp:752:22 #3 0x55cbb11eafc4 in DB::KeeperStateMachine::init() build_docker/./src/Coordination/KeeperStateMachine.cpp:91:69 #4 0x55cbb106e51c in DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) build_docker/./src/Coordination/KeeperServer.cpp:537:20 #5 0x55cbb1016813 in DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::__1::shared_ptr const&) build_docker/./src/Coordination/KeeperDispatcher.cpp:468:17 #6 0x55cba97fa016 in DB::Context::initializeKeeperDispatcher(bool) const build_docker/./src/Interpreters/Context.cpp:3439:36 #7 0x55cb9a60c51c in DB::Server::main(std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&) build_docker/./programs/server/Server.cpp:1609:25 #8 0x55cbb78b841e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 #9 0x55cb9a5e4e03 in DB::Server::run() build_docker/./programs/server/Server.cpp:418:25 #10 0x55cbb78fcd48 in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 #11 0x55cb9a5dce19 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:224:20 #12 0x55cb87e14e8a in main build_docker/./programs/main.cpp:498:21 #13 0x7f85f0703d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e) SUMMARY: AddressSanitizer: heap-use-after-free build_docker/./src/Coordination/SnapshotableHashTable.h:410:23 in DB::SnapshotableHashTable::disableSnapshotMode() Shadow bytes around the buggy address: 0x519000099100: 00 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x519000099280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x519000099380:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x519000099600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==816==ABORTING Logging test to /var/log/clickhouse-server/clickhouse-server.log Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:3076, shutdown) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] INFO : Connecting to 172.16.35.8(172.16.35.8):9181, use_ssl: False (connection.py:650, _connect) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/logs/stderr.log* || true'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/.env', '--project-name', 'roottestkeeperforcerecovery', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node5/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node6/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node7/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_force_recovery/_instances_0/node8/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node3_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node6_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node1_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node4_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node2_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node5_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node8_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node7_1 ... (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node7_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node6_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node8_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node5_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node3_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node2_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node1_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing roottestkeeperforcerecovery_node4_1 ... done (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stderr:Removing network roottestkeeperforcerecovery_default (cluster.py:123, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Cleanup called (cluster.py:801, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Docker networks for project roottestkeeperforcerecovery are NETWORK ID NAME DRIVER SCOPE (cluster.py:780, print_all_docker_pieces) 2025-06-05 19:55:47 [ 527 ] DEBUG : Docker containers for project roottestkeeperforcerecovery are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:788, print_all_docker_pieces) 2025-06-05 19:55:47 [ 527 ] DEBUG : Docker volumes for project roottestkeeperforcerecovery are DRIVER VOLUME NAME (cluster.py:796, print_all_docker_pieces) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperforcerecovery_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Unstopped containers: {} (cluster.py:815, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : No running containers for project: roottestkeeperforcerecovery (cluster.py:829, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Trying to prune unused networks... (cluster.py:835, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Trying to prune unused images... (cluster.py:851, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2025-06-05 19:55:47 [ 527 ] DEBUG : Images pruned (cluster.py:854, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Trying to prune unused volumes... (cluster.py:860, cleanup) 2025-06-05 19:55:47 [ 527 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2025-06-05 19:55:48 [ 527 ] DEBUG : Stdout:3 (cluster.py:121, run_and_check) ============================== slowest durations =============================== 305.63s call test_lost_part/test.py::test_lost_part_other_replica 107.09s call test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 94.32s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 80.24s call test_keeper_force_recovery/test.py::test_cluster_recovery 79.87s setup test_keeper_force_recovery/test.py::test_cluster_recovery 71.60s call test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive 65.80s call test_keeper_internal_secure/test.py::test_secure_raft_works_with_password 62.76s setup test_jbod_balancer/test.py::test_jbod_balanced_merge 61.49s setup test_keeper_s3_snapshot/test.py::test_s3_upload 57.51s call test_lost_part/test.py::test_lost_part_mutation 57.10s setup test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 55.42s call test_keeper_mntr_pressure/test.py::test_aggressive_mntr 54.04s setup test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] 53.39s setup test_lost_part/test.py::test_lost_last_part 52.80s setup test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down 51.98s setup test_fix_metadata_version/test.py::test_fix_metadata_version 50.95s setup test_keeper_availability_zone/test.py::test_get_availability_zone 50.77s setup test_log_family_s3/test.py::test_log_family_s3[Log] 49.36s setup test_keeper_four_word_command/test.py::test_cmd_clrs 48.96s setup test_failed_async_inserts/test.py::test_failed_async_inserts 48.83s setup test_library_bridge/test.py::test_load_all 48.50s setup test_keeper_persistent_log/test.py::test_ephemeral_after_restart 48.47s setup test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 47.57s setup test_keeper_four_word_command/test_allow_list.py::test_allow_list 47.34s setup test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 47.09s setup test_grant_and_revoke/test.py::test_allowed_grantees 46.89s setup test_jbod_ha/test.py::test_jbod_ha 46.85s setup test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used 46.67s setup test_keeper_internal_secure/test.py::test_secure_raft_works 46.11s setup test_drop_if_empty/test.py::test_drop_if_empty 44.90s setup test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper 43.63s setup test_freeze_table/test.py::test_freeze_table 42.97s setup test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 41.97s setup test_keeper_session/test.py::test_session_close_shutdown 41.90s setup test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local 41.54s setup test_keeper_reconfig_add/test.py::test_reconfig_add 41.25s setup test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 41.18s setup test_group_array_element_size/test.py::test_max_exement_size 40.58s call test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 40.03s setup test_http_native/test.py::test_http_native_returns_timezone 39.91s call test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 39.20s setup test_keeper_secure_client/test.py::test_connection 38.73s setup test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart 38.56s call test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down 38.45s setup test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view 37.92s setup test_fetch_memory_usage/test.py::test_huge_column 36.42s setup test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts 36.38s setup test_keeper_and_access_storage/test.py::test_create_replicated 35.38s setup test_keeper_nodes_move/test.py::test_node_move 33.65s call test_lost_part/test.py::test_lost_last_part 33.41s call test_keeper_nodes_move/test.py::test_node_move 32.52s setup test_keeper_mntr_pressure/test.py::test_aggressive_mntr 30.89s call test_jbod_ha/test.py::test_jbod_ha 25.27s call test_keeper_s3_snapshot/test.py::test_s3_upload 22.56s teardown test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 22.30s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog] 22.23s teardown test_keeper_s3_snapshot/test.py::test_s3_upload 19.42s call test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 18.51s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 17.11s call test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart 16.76s call test_group_array_element_size/test.py::test_max_exement_size 16.00s call test_keeper_internal_secure/test.py::test_secure_raft_works 14.94s call test_jbod_balancer/test.py::test_jbod_balanced_merge 14.37s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 13.85s teardown test_keeper_reconfig_add/test.py::test_reconfig_add 13.74s call test_keeper_session/test.py::test_session_close_shutdown 12.68s call test_library_bridge/test.py::test_recover_after_bridge_crash 11.85s call test_library_bridge/test.py::test_load_all 11.83s teardown test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts 10.79s call test_keeper_persistent_log/test.py::test_state_duplicate_restart 10.65s call test_memory_limit_observer/test.py::test_observe_memory_limit 9.15s call test_keeper_reconfig_add/test.py::test_reconfig_add 8.58s teardown test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view 8.57s call test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper 8.14s call test_library_bridge/test.py::test_load_all_many_rows 8.10s call test_fix_metadata_version/test.py::test_fix_metadata_version 7.36s call test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 6.97s call test_lost_part/test.py::test_lost_part_same_replica 6.70s call test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart 6.25s call test_grant_and_revoke/test.py::test_revoke_requires_grant_option 5.71s teardown test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 5.65s teardown test_lost_part/test.py::test_lost_part_same_replica 5.63s call test_grant_and_revoke/test.py::test_implicit_show_grants 5.33s teardown test_fetch_memory_usage/test.py::test_huge_column 5.29s teardown test_failed_async_inserts/test.py::test_failed_async_inserts 5.15s teardown test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table 5.08s teardown test_keeper_secure_client/test.py::test_connection 4.85s call test_keeper_persistent_log/test.py::test_state_after_restart 4.84s call test_keeper_persistent_log/test.py::test_ephemeral_after_restart 4.67s teardown test_drop_if_empty/test.py::test_drop_if_empty 4.65s teardown test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart 4.59s call test_grant_and_revoke/test.py::test_introspection 4.33s call test_drop_if_empty/test.py::test_drop_if_empty 4.30s teardown test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local 4.29s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 4.25s call test_grant_and_revoke/test.py::test_allowed_grantees 4.13s teardown test_grant_and_revoke/test.py::test_smoke 4.12s teardown test_memory_limit_observer/test.py::test_observe_memory_limit 4.03s call test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke 3.94s teardown test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin 3.89s call test_grant_and_revoke/test.py::test_grant_with_replace_option 3.85s teardown test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 3.82s teardown test_http_native/test.py::test_http_native_returns_timezone 3.33s teardown test_jbod_ha/test.py::test_jbod_ha 3.20s teardown test_freeze_table/test.py::test_freeze_table 3.12s teardown test_keeper_availability_zone/test.py::test_get_availability_zone 3.05s teardown test_keeper_session/test.py::test_session_timeout 2.96s teardown test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper 2.84s teardown test_keeper_force_recovery/test.py::test_cluster_recovery 2.80s teardown test_fix_metadata_version/test.py::test_fix_metadata_version 2.67s call test_grant_and_revoke/test.py::test_implicit_create_view_grant 2.53s call test_grant_and_revoke/test.py::test_current_grants_override 2.51s teardown test_keeper_nodes_move/test.py::test_node_move 2.44s teardown test_keeper_mntr_pressure/test.py::test_aggressive_mntr 2.42s teardown test_keeper_four_word_command/test_allow_list.py::test_allow_list 2.34s teardown test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 2.30s call test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] 2.30s call test_grant_and_revoke/test.py::test_grant_current_grants 2.18s teardown test_keeper_internal_secure/test.py::test_secure_raft_works_with_password 2.14s teardown test_keeper_four_word_command/test.py::test_cmd_ydld 2.13s call test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] 1.94s call test_grant_and_revoke/test.py::test_grant_option 1.93s teardown test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart 1.89s teardown test_group_array_element_size/test.py::test_max_exement_size 1.87s teardown test_library_bridge/test.py::test_ssrf 1.82s teardown test_keeper_persistent_log/test.py::test_state_duplicate_restart 1.75s call test_log_family_s3/test.py::test_log_family_s3[TinyLog] 1.75s teardown test_keeper_and_access_storage/test.py::test_create_replicated 1.74s call test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts 1.67s teardown test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 1.67s call test_keeper_four_word_command/test_allow_list.py::test_allow_list 1.63s call test_insert_over_http_query_log/test.py::test_insert_over_http_ok 1.61s call test_log_family_s3/test.py::test_log_family_s3[Log] 1.56s call test_library_bridge/test.py::test_path_validation 1.56s call test_library_bridge/test.py::test_load_ids 1.56s call test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect 1.55s call test_log_family_s3/test.py::test_log_family_s3[StripeLog] 1.49s call test_freeze_table/test.py::test_freeze_table 1.46s call test_failed_async_inserts/test.py::test_failed_async_inserts 1.44s call test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes 1.42s call test_grant_and_revoke/test.py::test_current_database 1.30s call test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk 1.29s call test_library_bridge/test.py::test_load_keys 1.24s call test_grant_and_revoke/test.py::test_grant_all_on_table 1.04s call test_grant_and_revoke/test.py::test_smoke 1.04s call test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant 1.04s call test_keeper_four_word_command/test.py::test_cmd_ydld 1.03s call test_keeper_four_word_command/test.py::test_cmd_rqld 0.95s call test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view 0.93s call test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local 0.84s call test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table 0.84s call test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement 0.76s call test_library_bridge/test.py::test_null_values 0.76s call test_keeper_four_word_command/test.py::test_cmd_mntr 0.61s call test_keeper_four_word_command/test.py::test_cmd_lgif 0.55s call test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used 0.50s call test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin 0.50s call test_fetch_memory_usage/test.py::test_huge_column 0.38s teardown test_grant_and_revoke/test.py::test_allowed_grantees 0.38s teardown test_grant_and_revoke/test.py::test_grant_option 0.36s call test_keeper_four_word_command/test.py::test_cmd_crst 0.34s call test_keeper_availability_zone/test.py::test_get_availability_zone 0.33s call test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 0.33s teardown test_grant_and_revoke/test.py::test_implicit_create_view_grant 0.33s teardown test_grant_and_revoke/test.py::test_introspection 0.33s teardown test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant 0.33s teardown test_grant_and_revoke/test.py::test_revoke_requires_grant_option 0.33s teardown test_grant_and_revoke/test.py::test_grant_with_replace_option 0.33s teardown test_grant_and_revoke/test.py::test_grant_all_on_table 0.33s teardown test_grant_and_revoke/test.py::test_current_grants_override 0.33s teardown test_grant_and_revoke/test.py::test_grant_current_grants 0.32s call test_keeper_four_word_command/test.py::test_cmd_stat 0.29s call test_keeper_four_word_command/test.py::test_cmd_dump 0.28s teardown test_grant_and_revoke/test.py::test_implicit_show_grants 0.28s teardown test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke 0.28s teardown test_grant_and_revoke/test.py::test_current_database 0.28s call test_keeper_four_word_command/test.py::test_cmd_wchs 0.27s call test_keeper_four_word_command/test.py::test_cmd_cons 0.27s call test_keeper_four_word_command/test.py::test_cmd_wchc 0.26s call test_keeper_four_word_command/test.py::test_cmd_srvr 0.26s call test_keeper_four_word_command/test.py::test_cmd_wchp 0.22s call test_keeper_secure_client/test.py::test_connection 0.22s call test_keeper_four_word_command/test.py::test_cmd_clrs 0.17s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 0.17s call test_keeper_and_access_storage/test.py::test_create_replicated 0.17s call test_library_bridge/test.py::test_ssrf 0.15s call test_keeper_four_word_command/test.py::test_cmd_srst 0.14s call test_keeper_four_word_command/test.py::test_cmd_conf 0.11s call test_keeper_four_word_command/test.py::test_cmd_csnp 0.09s call test_http_native/test.py::test_http_native_returns_timezone 0.04s call test_keeper_session/test.py::test_session_timeout 0.02s call test_keeper_four_word_command/test.py::test_cmd_isro 0.01s call test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s teardown test_jbod_balancer/test.py::test_jbod_balanced_merge 0.00s setup test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive 0.00s teardown test_lost_part/test.py::test_lost_last_part 0.00s teardown test_keeper_persistent_log/test.py::test_state_after_restart 0.00s setup test_grant_and_revoke/test.py::test_implicit_create_view_grant 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_isro 0.00s setup test_grant_and_revoke/test.py::test_smoke 0.00s setup test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ydld 0.00s setup test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s setup test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch 0.00s setup test_keeper_persistent_log/test.py::test_state_after_restart 0.00s setup test_grant_and_revoke/test.py::test_revoke_requires_grant_option 0.00s setup test_library_bridge/test.py::test_null_values 0.00s setup test_lost_part/test.py::test_lost_part_other_replica 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log] 0.00s setup test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin 0.00s setup test_grant_and_revoke/test.py::test_introspection 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart 0.00s setup test_grant_and_revoke/test.py::test_grant_with_replace_option 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_crst 0.00s teardown test_keeper_persistent_log/test.py::test_ephemeral_after_restart 0.00s teardown test_library_bridge/test.py::test_recover_after_bridge_crash 0.00s setup test_lost_part/test.py::test_lost_part_same_replica 0.00s setup test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s setup test_library_bridge/test.py::test_load_keys 0.00s setup test_library_bridge/test.py::test_load_all_many_rows 0.00s setup test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke 0.00s setup test_grant_and_revoke/test.py::test_implicit_show_grants 0.00s setup test_lost_part/test.py::test_lost_part_mutation 0.00s teardown test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk 0.00s setup test_library_bridge/test.py::test_load_ids 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_library_bridge/test.py::test_path_validation 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s setup test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_cons 0.00s setup test_keeper_four_word_command/test.py::test_cmd_cons 0.00s setup test_keeper_four_word_command/test.py::test_cmd_isro 0.00s setup test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes 0.00s teardown test_keeper_internal_secure/test.py::test_secure_raft_works 0.00s teardown test_library_bridge/test.py::test_null_values 0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 0.00s setup test_keeper_persistent_log/test.py::test_state_duplicate_restart 0.00s setup test_keeper_internal_secure/test.py::test_secure_raft_works_with_password 0.00s setup test_grant_and_revoke/test.py::test_grant_current_grants 0.00s setup test_keeper_session/test.py::test_session_timeout 0.00s setup test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement 0.00s teardown test_library_bridge/test.py::test_load_ids 0.00s teardown test_library_bridge/test.py::test_load_keys 0.00s setup test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s setup test_library_bridge/test.py::test_ssrf 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] 0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 0.00s teardown test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used 0.00s teardown test_library_bridge/test.py::test_load_all 0.00s teardown test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] 0.00s setup test_grant_and_revoke/test.py::test_grant_all_on_table 0.00s teardown test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes 0.00s teardown test_library_bridge/test.py::test_load_all_many_rows 0.00s teardown test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down 0.00s teardown test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive 0.00s setup test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect 0.00s setup test_grant_and_revoke/test.py::test_grant_option 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srst 0.00s setup test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_clrs 0.00s teardown test_lost_part/test.py::test_lost_part_other_replica 0.00s teardown test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect 0.00s setup test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant 0.00s setup test_grant_and_revoke/test.py::test_current_grants_override 0.00s setup test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_library_bridge/test.py::test_recover_after_bridge_crash 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_keeper_session/test.py::test_session_close_shutdown 0.00s setup test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 0.00s teardown test_lost_part/test.py::test_lost_part_mutation 0.00s setup test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_grant_and_revoke/test.py::test_current_database 0.00s setup test_insert_over_http_query_log/test.py::test_insert_over_http_ok 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_crst 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_keeper_four_word_command/test.py::test_cmd_stat 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 0.00s setup test_memory_limit_observer/test.py::test_observe_memory_limit 0.00s setup test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size 0.00s teardown test_library_bridge/test.py::test_path_validation 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srst 0.00s teardown test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement 0.00s teardown test_insert_over_http_query_log/test.py::test_insert_over_http_ok 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_ruok =========================== short test summary info ============================ ERROR test_keeper_force_recovery/test.py::test_cluster_recovery - Exception: ... PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes PASSED test_keeper_four_word_command/test.py::test_cmd_clrs PASSED test_keeper_four_word_command/test.py::test_cmd_conf PASSED test_keeper_four_word_command/test.py::test_cmd_cons PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk PASSED test_keeper_four_word_command/test.py::test_cmd_crst PASSED test_keeper_four_word_command/test.py::test_cmd_csnp PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin PASSED test_keeper_four_word_command/test.py::test_cmd_dump PASSED test_keeper_four_word_command/test.py::test_cmd_isro PASSED test_grant_and_revoke/test.py::test_allowed_grantees PASSED test_keeper_four_word_command/test.py::test_cmd_lgif PASSED test_keeper_four_word_command/test.py::test_cmd_mntr PASSED test_log_family_s3/test.py::test_log_family_s3[Log] PASSED test_grant_and_revoke/test.py::test_current_database PASSED test_keeper_four_word_command/test.py::test_cmd_rqld PASSED test_keeper_four_word_command/test.py::test_cmd_ruok PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart PASSED test_keeper_four_word_command/test.py::test_cmd_srst PASSED test_keeper_four_word_command/test.py::test_cmd_srvr PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] PASSED test_keeper_four_word_command/test.py::test_cmd_stat PASSED test_keeper_four_word_command/test.py::test_cmd_wchc PASSED test_keeper_four_word_command/test.py::test_cmd_wchp PASSED test_keeper_four_word_command/test.py::test_cmd_wchs PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] PASSED test_keeper_four_word_command/test.py::test_cmd_ydld PASSED test_grant_and_revoke/test.py::test_current_grants_override PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0] PASSED test_grant_and_revoke/test.py::test_grant_all_on_table PASSED test_keeper_persistent_log/test.py::test_state_after_restart PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1] PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement PASSED test_grant_and_revoke/test.py::test_grant_current_grants PASSED test_library_bridge/test.py::test_load_all PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_ok PASSED test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table PASSED test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke PASSED test_grant_and_revoke/test.py::test_grant_option PASSED test_library_bridge/test.py::test_load_all_many_rows PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart PASSED test_library_bridge/test.py::test_load_ids PASSED test_grant_and_revoke/test.py::test_grant_with_replace_option PASSED test_library_bridge/test.py::test_load_keys PASSED test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant PASSED test_library_bridge/test.py::test_null_values PASSED test_library_bridge/test.py::test_path_validation PASSED test_grant_and_revoke/test.py::test_implicit_create_view_grant PASSED test_grant_and_revoke/test.py::test_implicit_show_grants PASSED test_grant_and_revoke/test.py::test_introspection PASSED test_library_bridge/test.py::test_recover_after_bridge_crash PASSED test_lost_part/test.py::test_lost_last_part PASSED test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down PASSED test_grant_and_revoke/test.py::test_revoke_requires_grant_option PASSED test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect PASSED test_grant_and_revoke/test.py::test_smoke PASSED test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts PASSED test_failed_async_inserts/test.py::test_failed_async_inserts PASSED test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size PASSED test_keeper_session/test.py::test_session_close_shutdown PASSED test_keeper_session/test.py::test_session_timeout PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted PASSED test_memory_limit_observer/test.py::test_observe_memory_limit PASSED test_jbod_balancer/test.py::test_jbod_balanced_merge PASSED test_keeper_internal_secure/test.py::test_secure_raft_works PASSED test_lost_part/test.py::test_lost_part_mutation PASSED test_http_native/test.py::test_http_native_returns_timezone PASSED test_group_array_element_size/test.py::test_max_exement_size PASSED test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive PASSED test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch PASSED test_keeper_availability_zone/test.py::test_get_availability_zone PASSED test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper PASSED test_fix_metadata_version/test.py::test_fix_metadata_version PASSED test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local PASSED test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts PASSED test_keeper_internal_secure/test.py::test_secure_raft_works_with_password PASSED test_drop_if_empty/test.py::test_drop_if_empty PASSED test_jbod_ha/test.py::test_jbod_ha PASSED test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart PASSED test_freeze_table/test.py::test_freeze_table PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr PASSED test_keeper_and_access_storage/test.py::test_create_replicated PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 PASSED test_keeper_secure_client/test.py::test_connection PASSED test_keeper_nodes_move/test.py::test_node_move PASSED test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 PASSED test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view PASSED test_keeper_reconfig_add/test.py::test_reconfig_add PASSED test_keeper_s3_snapshot/test.py::test_s3_upload PASSED test_keeper_force_recovery/test.py::test_cluster_recovery PASSED test_lost_part/test.py::test_lost_part_other_replica PASSED test_lost_part/test.py::test_lost_part_same_replica SKIPPED [1] test_merge_tree_load_parts/test.py:227: Skip with debug build and sanitizers. This test intentionally triggers LOGICAL_ERROR which leads to crash with those builds SKIPPED [1] test_library_bridge/test.py:338: Sanitizer cannot work with third-party shared libraries SKIPPED [1] test_fetch_memory_usage/test.py:26: sanitizer build has higher memory consumption; also it is slow SKIPPED [1] test_library_bridge/test_exiled.py:53: Leak sanitizer falsely reports about a leak of 16 bytes in clickhouse-odbc-bridge ============== 96 passed, 4 skipped, 1 error in 513.39s (0:08:33) ============== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 437, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_rc7sjw --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=2993bc2bf171 -e DOCKER_KERBERIZED_HADOOP_TAG=ce74919e88f5 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=a2d3dc777d0c -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_drop_if_empty/test.py::test_drop_if_empty test_drop_replica_with_auxiliary_zookeepers/test.py::test_drop_replica_in_auxiliary_zookeeper test_failed_async_inserts/test.py::test_failed_async_inserts test_fetch_memory_usage/test.py::test_huge_column test_fetch_partition_with_outdated_parts/test.py::test_fetch_partition_with_outdated_parts test_fix_metadata_version/test.py::test_fix_metadata_version test_freeze_table/test.py::test_freeze_table test_grant_and_revoke/test.py::test_allowed_grantees test_grant_and_revoke/test.py::test_current_database test_grant_and_revoke/test.py::test_current_grants_override test_grant_and_revoke/test.py::test_grant_all_on_table test_grant_and_revoke/test.py::test_grant_current_grants test_grant_and_revoke/test.py::test_grant_current_grants_with_partial_revoke test_grant_and_revoke/test.py::test_grant_option test_grant_and_revoke/test.py::test_grant_with_replace_option test_grant_and_revoke/test.py::test_implicit_create_temporary_table_grant test_grant_and_revoke/test.py::test_implicit_create_view_grant test_grant_and_revoke/test.py::test_implicit_show_grants test_grant_and_revoke/test.py::test_introspection test_grant_and_revoke/test.py::test_revoke_requires_grant_option test_grant_and_revoke/test.py::test_smoke test_group_array_element_size/test.py::test_max_exement_size test_host_regexp_multiple_ptr_records_concurrent/test.py::test_host_regexp_multiple_ptr_v4 test_http_native/test.py::test_http_native_returns_timezone test_insert_into_distributed_through_materialized_view/test.py::test_inserts_local 'test_insert_over_http_query_log/test.py::test_insert_over_http_exception[0]' 'test_insert_over_http_query_log/test.py::test_insert_over_http_exception[1]' test_insert_over_http_query_log/test.py::test_insert_over_http_invalid_statement test_insert_over_http_query_log/test.py::test_insert_over_http_ok test_insert_over_http_query_log/test.py::test_insert_over_http_unknown_table test_inserts_with_keeper_retries/test.py::test_query_timeout_with_zk_down test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_disconnect test_inserts_with_keeper_retries/test.py::test_replica_inserts_with_keeper_restart test_jbod_balancer/test.py::test_jbod_balanced_merge test_jbod_balancer/test.py::test_replicated_balanced_merge_fetch test_jbod_ha/test.py::test_jbod_ha test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_detect_background_changes test_jbod_load_balancing/test.py::test_jbod_load_balancing_least_used_next_disk test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_availability_zone/test.py::test_get_availability_zone test_keeper_force_recovery/test.py::test_cluster_recovery test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test.py::test_cmd_ydld test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_internal_secure/test.py::test_secure_raft_works test_keeper_internal_secure/test.py::test_secure_raft_works_with_password test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_reconfig_add/test.py::test_reconfig_add test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_s3_snapshot/test.py::test_s3_upload test_keeper_secure_client/test.py::test_connection test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_library_bridge/test.py::test_load_all test_library_bridge/test.py::test_load_all_many_rows test_library_bridge/test.py::test_load_ids test_library_bridge/test.py::test_load_keys test_library_bridge/test.py::test_null_values test_library_bridge/test.py::test_path_validation test_library_bridge/test.py::test_recover_after_bridge_crash test_library_bridge/test.py::test_server_restart_bridge_might_be_still_alive test_library_bridge/test.py::test_ssrf test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_lost_part/test.py::test_lost_last_part test_lost_part/test.py::test_lost_part_mutation test_lost_part/test.py::test_lost_part_other_replica test_lost_part/test.py::test_lost_part_same_replica test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view test_memory_limit_observer/test.py::test_memory_usage_doesnt_include_page_cache_size test_memory_limit_observer/test.py::test_observe_memory_limit test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error -vvv" altinityinfra/integration-tests-runner:9d492c2eec24 ' returned non-zero exit status 1.